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SUBJECT:  Report  on  Hurricane  Katrina  Disaster  Recovery  Efforts  Related  to  Army 
Information  Technology  Resources  (Report  No.  D-2007-006) 


We  are  providing  this  report  for  information  and  use.  We  considered 
management  comments  on  a  draft  of  this  report  in  preparing  the  final  report. 

Comments  on  the  draft  of  this  report  conformed  to  the  requirements  of  DoD 
Directive  7650.3  and  left  no  unresolved  issues.  Therefore,  no  additional  comments  are 
required. 

We  appreciate  the  courtesies  extended  to  the  staff.  Questions  should  be  directed 
to  Ms.  Jacqueline  Wicecarver  at  (703)  604-9077  (DSN  664-9077)  or  Ms.  Therese  M. 
Kince  at  (703)  604-9060  (DSN  664-9060).  The  team  members  are  listed  inside  the  back 
cover. 


By  direction  of  the  Deputy  Inspector  General  for  Auditing: 


Bruce  A.  Burton 

Deputy  Assistant  Inspector  General 
Acquisition  and  Contract  Management 


Department  of  Defense  Office  of  Inspector  General 


Report  No.  D2007-006  October  19,  2006 
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Hurricane  Katrina  Disaster  Recovery  Efforts  Related  to 
Army  Information  Technology  Resources 


Executive  Summary 


Who  Should  Read  This  Report  and  Why?  Managers  of  DoD  information  systems 
should  read  this  report  because  it  emphasizes  the  importance  of  continuity  of  operations 
planning  for  critical  systems  that  may  be  disrupted  during  disasters. 

Background.  On  August  29,  2005,  Hurricane  Katrina  devastated  the  Gulf  Coast  states 
of  Louisiana,  Mississippi,  Alabama,  and  Florida  with  Category  3  winds  and  torrential 
rain.  This  audit  report  is  the  first  in  a  planned  series  of  audits  on  the  effects  of  Hurricane 
Katrina  on  DoD  information  technology  resources.  Two  Army  logistics  information 
technology  systems  operated  by  the  321st  Theater  Materiel  Management  Center  Rear, 
located  in  Baton  Rouge,  Louisiana,  experienced  communications  disruptions  as  a  result 
of  Hurricane  Katrina. 

The  Army  information  technology  resources  affected  by  Hurricane  Katrina  in  Baton 
Rouge  were  categorized  as  Mission  Assurance  Category  II  systems.  DoD  Instruction 
8500.2,  “Information  Assurance  Implementation,”  February  6,  2003,  requires  Mission 
Assurance  Category  II  systems  to  have  a  disaster  plan  that  enables  mission-  or  business- 
essential  functions  to  resume  within  24  hours.  DoD  Directive  3020.26,  “Defense 
Continuity  Plan,”  September  8,  2004,  requires  DoD  Components  to  develop,  coordinate, 
and  maintain  continuity  plans;  to  update  and  reissue  plans  every  2  years;  and  to  test  and 
exercise  continuity  plans  at  least  annually,  or  otherwise  as  directed. 

Results.  The  321st  Theater  Materiel  Management  Center  Rear  lost  voice  and  data 
communications  as  a  result  of  Hurricane  Katrina.  The  materiel  managers  of  the 
321st  Theater  Materiel  Management  Center  Rear  were  unable  to  communicate  with 
customers  located  in  Southwest  Asia,  nor  could  they  resume  operations  for  one  critical 
logistics  infonnation  technology  system  within  24  hours.  As  a  result,  a  backlog  in 
processing  requisitions  caused  a  4-day  delay  in  requisitioning  supplies  from  warehouses; 
the  materiel  managers  of  the  321st  Theater  Materiel  Management  Center  Rear  could  not 
quantify  the  impact  on  the  warfighter  or  theater  readiness.  After  Hurricane  Katrina,  the 
321st  Theater  Materiel  Management  Center  Rear  took  the  appropriate  steps  to  correct 
disaster  recovery  planning  weaknesses.  According  to  officials  at  the  Army  Materiel 
Command  and  the  377th  Theater  Support  Command,  the  two  logistics  information 
technology  systems  will  transfer  from  the  321st  Theater  Materiel  Management  Center 
Rear  to  the  Army  Materiel  Command  Logistics  Support  Activity,  Redstone  Arsenal, 
Alabama,  and  Belle  Chasse,  Louisiana,  respectively.  (See  the  Finding  section  of  the 
report  for  the  detailed  recommendations.)  The  321st  Theater  Materiel  Management 
Center  Rear  internal  controls  were  not  adequate.  We  identified  an  internal  control 
weakness  in  the  protection  of  infonnation  technology  resources. 


Management  Comments.  The  U.  S.  Army  Materiel  Command  Director,  Internal 
Review  and  Audit  Compliance  Office,  responding  for  the  Command,  stated  that 
Headquarters  had  reviewed  the  draft  report  and  agreed  with  the  reported  facts, 
conclusions,  and  concurred  with  the  recommendations.  U.S.  Army  Materiel  Command 
agreed  to  develop  a  disaster  recovery  plan  for  the  Corps  Theater  Automation  Service 
Support  Center  in  accordance  with  DoD  Instruction  8500.2  and  DoD  Directive  3020.26 
by  January  1,  2007,  and  stated  the  plan  will  be  exercised  at  least  annually  and  updated 
semiannually. 

The  Commanding  General  of  the  U.S.  Army  377th  Theater  Support  Command  concurred 
with  comments.  The  Commanding  General  stated  that  the  Command  recognized  and 
corrected  the  deficiencies  prior  to  the  audit.  After  Hurricane  Katrina,  the  Command 
developed  and  tested  a  disaster  recovery  plan.  They  added  redundant  data 
communications  and  trained  staff  on  this  process.  Additionally,  the  Commanding 
General  stated  the  Standard  Army  Ammunition  System-Modernized  was  transferred  to 
Belle  Chasse  on  July  12,  2006,  and  the  Command  was  in  the  process  of  modifying  the 
disaster  recovery  plan  to  include  the  system  at  that  location.  The  plan  will  include 
disaster  recovery  measures  that  comply  with  Federal  and  DoD  policy  at  the  final 
locations  of  both  the  primary  and  back-up  systems  for  the  Standard  Army  Ammunition 
System-Modernized.  Finally,  the  Commanding  General  stated  that  the  new  plan  will  be 
completed  by  November  30,  2006.  See  the  Finding  section  of  the  report  for  a  discussion 
of  management  comments  and  the  Management  Comments  section  of  the  report  for  the 
complete  text  of  the  comments. 
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Background 


On  August  29,  2005,  Hurricane  Katrina  devastated  the  Gulf  Coast  states  of 
Louisiana,  Mississippi,  Alabama,  and  Florida  with  Category  3  winds  and 
torrential  rain.  This  audit  report  is  the  first  in  a  planned  series  of  audits  on  the 
effects  of  Hurricane  Katrina  on  DoD  information  technology  resources. 

Mission.  The  321st  Theater  Materiel  Management  Center  is  a  U.S.  Army 
Reserve  unit  under  the  377th  Theater  Support  Command.  The  321st  Theater 
Materiel  Management  Center  Rear,1  located  in  Baton  Rouge,  Louisiana,  is 
responsible  for: 

•  integrating  joint  systems  analysis  and  supply  and  maintenance 
management  in  the  Central  Command  Area  of  Responsibility; 

•  directing  and  synchronizing  materiel  management  organizations  and 
supporting  the  Materiel  Management  Center’s  Standard  Anny 
Management  Infonnation  System; 

•  supervising  Rear  Elements  and  coordinating  and  tasking  authority  over 
Forward2  Deployed  Elements;  and 

•  managing  the  supply  system  to  identify  the  status  and  location  of 
materiel  into,  through,  and  out  of  Central  Command  Area  of 
Responsibility. 

The  321st  Theater  Materiel  Management  Center  Rear  is  an  operator  of  the  Corps 
Theater  Automatic  Data  Processing  Service  Center  (CTASC)  and  the  Standard 
Anny  Ammunition  System-Modernized,  both  Mission  Assurance  Category  II 
systems,  which  are  used  to  manage  supply  operations  in  Southwest  Asia. 

The  Systems.  The  CTASC  system  is  decentralized  throughout  the  world  at 
nine  locations.  In  Baton  Rouge,  the  CTASC  manages  supply  transactions  for 
several  warehouses  in  Southwest  Asia,  and  the  Department  of  the  Army 
designated  it  as  the  most  critical  system  in  the  world  because  it  manages  all 
requisitions  within  Southwest  Asia,  including  Iraq,  Afghanistan,  and  the  Horn  of 
Africa.  The  CTASC  provides  direct  support  to  the  warfighter  by  processing 
materiel  requisitions  from  52  warehouses.  The  321st  Theater  Materiel 
Management  Center  Rear  also  operates  the  Standard  Army  Ammunition  System- 
Modernized  to  manage  the  ammunition  supply  for  and  within  Southwest  Asia. 

The  Standard  Anny  Ammunition  System-Modernized  is  a  portable  system  based 
on  commercial  software  with  flexible  communications  capabilities. 


1  The  321st  Theater  Material  Management  Center  Rear,  in  Baton  Rouge,  Louisiana,  provides  direct  support 
to  Iraq,  Kuwait,  Afghanistan,  and  the  Horn  of  Africa. 

2  The  321st  Theater  Materiel  Management  Center  Forward,  in  Camp  Arifjan,  Kuwait,  coordinates  with  the 
Rear  in  managing  the  distribution  and  accountability  of  ammunitions  in  the  Central  Command  Area  of 
Responsibility. 
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Criteria 


All  DoD  organizations  are  required  to  comply  with  the  following  laws  and 
regulations  when  they  implement  their  disaster  recovery  controls  and  plans. 

Information  Assurance.  DoD  Directive  8500.1,  “Information  Assurance,” 
October  24,  2002,  certified  current  as  of  November  21,  2003,  states  that  Mission 
Assurance  Category  II  systems  handle  information  that  is  important  to  support 
deployed  and  contingency  forces.  The  loss  of  a  Mission  Assurance  Category  II 
system  is  difficult  to  deal  with  and  can  be  tolerated  only  for  a  short  time.  The 
consequences  of  a  delay  in  providing  important  information  or  of  a  degradation  of 
services  may  seriously  affect  the  mission  and  operational  readiness.  Finally,  DoD 
Directive  8500.1  states  that  Mission  Assurance  Category  II  systems  require  other 
safeguards  in  addition  to  best  business  practices  to  provide  adequate  assurance. 

DoD  Instruction  8500.2,  “Information  Assurance  Implementation,”  February  6, 
2003,  establishes  a  baseline  for  information  assurance  with  a  list  of  threshold 
controls  to  maintain  the  availability  of  Mission  Assurance  Category  II  systems. 
These  controls  should  include  a  disaster  plan  that  will  resume  mission-  or 
business-essential  functions  within  24  hours  of  activation.  The  plan  should 
include  business  recovery  plans,  system  contingency  plans,  facility  disaster 
recovery  plans,  and  plan  acceptance.  The  controls  should  also  include  a 
continuity  of  operations  plan  or  a  disaster  recovery  plan  that  is  tested  and 
exercised  annually. 

Disaster  and  Recovery  Planning.  DoD  Directive  3020.26,  “Defense  Continuity 
Plan,”  September  8,  2004,  requires  a  comprehensive  and  effective  program  that 
continues  DoD  Components’  mission-essential  functions  under  all  circumstances 
and  threats.  Performing  mission-essential  functions  during  a  threat  or  event  must 
be  the  basis  for  continuity  planning,  preparation,  and  execution.  DoD  Directive 
3020.26  requires  the  heads  of  DoD  Components  to  develop,  coordinate,  and 
maintain  continuity  plans  and  to  update  and  reissue  plans  every  2  years.  Finally, 
the  heads  of  DoD  Components  should  test  and  exercise  continuity  plans  at  least 
annually,  or  otherwise  as  directed. 


Objective 


The  overall  audit  objective  was  to  determine  whether  adequate  internal  controls 
and  plans  for  disaster  recovery  were  in  place  to  safeguard  Army  information 
technology  resources.  Specifically,  we  reviewed  the  Army  infonnation 
technology  resources  managed  by  the  32 1st  Theater  Materiel  Management  Center 
Rear,  which  were  affected  by  Hurricane  Katrina. 
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Review  of  Internal  Controls 


We  identified  internal  control  weaknesses  for  the  321st  Theater  Materiel 
Management  Center  Rear  as  defined  by  DoD  Instruction  5010.40,  “Managers’ 
Internal  Control  (MIC)  Program  Procedures,”  January  4,  2006.  The  321st  Theater 
Materiel  Management  Center  Rear  in  Baton  Rogue,  Louisiana,  did  not  have 
procedures  in  place  to  adequately  protect  and  safeguard  infonnation  technology 
resources.  Implementing  Recommendations  1.  and  2.  will  improve  the  U.S.  Army 
Logistics  Support  Activity  (acquiring  the  CTASC  mission  from  the  321st  Theater 
Materiel  Management  Center  Rear)  and  377th  Theater  Support  Command 
planned  disaster  recovery  measures  for  continuity  of  operations  that  comply  with 
Federal  and  DoD  policy.  A  copy  of  the  final  report  will  be  provided  to  the  senior 
official  responsible  for  internal  controls  in  the  377th  Theater  Support  Command. 
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Effects  of  Hurricane  Katrina  on  Army 
Information  Technology  Resources 

The  321st  Theater  Materiel  Management  Center  Rear  lacked  internal 
controls  and  a  disaster  recovery  plan  to  adequately  protect  and  safeguard 
information  technology  resources.  Hurricane  Katrina  damaged  critical 
communications  circuits  that  were  used  by  supply  managers  at  the  321st 
Theater  Materiel  Management  Center  Rear  to  communicate  with 
customers  located  in  Southwest  Asia,  and  operations  were  not  resumed  for 
one  critical  logistics  information  technology  system  within  the  required 
24  hours.  These  conditions  occurred  because  the  321st  Theater  Materiel 
Management  Center  Rear  did  not  have  a  formal  disaster  recovery  plan  to 
continue  operations.  As  a  result,  a  backlog  in  processing  requisitions 
caused  a  4-day  delay  in  requisitioning  supplies  from  warehouses;  the 
supply  managers  at  the  321st  Theater  Materiel  Management  Center  Rear 
could  not  determine  the  impact  on  the  warfighter  or  on  theater  readiness. 


Communications  Outages 


The  321st  Theater  Materiel  Management  Center  Rear  lost  Defense  Switched 
Network  and  commercial  voice  and  data  communications  because  of  damage  to 
telecommunications  lines  caused  by  Hurricane  Katrina.  As  a  result,  the  321st 
Theater  Materiel  Management  Center  Rear  was  not  able  to  communicate  with  the 
users  of  the  CTASC  system  and  the  Standard  Anny  Ammunition  System- 
Modernized  in  Southwest  Asia. 

The  321st  Theater  Materiel  Management  Center  Rear  did  not  have  a  formal 
disaster  recovery  plan  in  place  prior  to  Hurricane  Katrina;  however,  it  worked 
with  a  multiorganization  crisis  action  team  to  bring  operations  back  online  using 
solutions  developed  as  the  situation  unfolded.  The  321st  Theater  Materiel 
Management  Center  Rear  and  the  crisis  action  team  were  able  to  reestablish 
communications  with  the  CTASC  system  customers  in  Southwest  Asia  within 
56  hours  of  the  initial  outage  by  using  satellite  communications. 

In  an  effort  to  maintain  communication  with  the  Standard  Army  Ammunition 
System-Modernized,  the  Army  officer  in  charge,  responsible  for  the  Standard 
Anny  Ammunition  System-Modernized,  asked  all  reporting  ammunition  supply 
points  to  communicate  by  e-mail.  The  Standard  Army  Ammunition  System- 
Modernized  operations  were  restored  within  24  hours  of  the  initial  outage  by 
using  an  e-mail  workaround  that  allowed  direct  movement  of  all  munitions  within 
the  Central  Command  Area  of  Responsibility  to  continue. 


Disaster  Recovery  Planning 


The  321st  Theater  Materiel  Management  Center  Rear  command  staff  stated  that 
they  discussed  several  disaster  recovery  options  before  Hurricane  Katrina,  such  as 
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obtaining  dual-homed  (T 1)  communication  lines  and  using  satellite 
communications  during  an  outage.  However,  these  options  were  not  formally 
documented  because  the  321st  Theater  Materiel  Management  Center  Rear 
command  staff  believed  that  they  did  not  have  the  resources  to  develop  a 
complete  disaster  recovery  plan.  The  321st  Theater  Materiel  Management  Center 
Rear  had  a  generic  continuity  of  operations  (COOP)  plan  for  the  CTASC  system 
only;  this  plan  was  limited  in  scope,  was  not  site-specific,  and  was  not  tested 
annually.  Although  the  321st  Theater  Materiel  Management  Center  Rear  took 
preliminary  steps  to  prepare  for  potential  disasters  prior  to  Hurricane  Katrina,  it 
lacked  a  complete  disaster  recovery  plan  for  all  its  critical  systems. 

The  321st  Theater  Materiel  Management  Center  Rear  did  not  have  a 
comprehensive  and  effective  disaster  recovery  program  before  Hurricane  Katrina 
as  required  by  DoD  Directive  3020.26  because  it  did  not  have  a  plan  that  would 
make  all  mission-essential  functions  available  under  all  circumstances  and  threats. 
Additionally,  due  to  the  lack  of  a  disaster  recovery  plan,  the  321st  Theater 
Materiel  Management  Center  Rear  was  not  able  to  resume  all  mission-essential 
functions  for  CTASC  operations  within  24  hours  or  exercise  the  generic  COOP 
plan  annually  as  required  for  Mission  Assurance  Category  II  systems  by  DoD 
Instruction  8500.2. 


Impact  to  Theater 


The  communications  outages  caused  a  backlog  of  batch  file  traffic,  resulting  in  a 
4-day  backlog  of  parts  requisitions  from  supply  support  organizations.  The  batch 
file  traffic  backlog  was  slowly  reduced  once  satellite  communications  were 
established  56  hours  after  Hurricane  Katrina  struck.  Communications  using 
T-l  telecommunications  lines  were  reestablished  within  13  days  and  the  backlog 
was  dramatically  reduced.  An  after-action  report  and  statements  made  by  321st 
Theater  Materiel  Management  Center  Rear  personnel  indicate  that  although  the 
total  impact  to  theater  readiness  is  unknown,  no  major  supply  problems  were 
evident  during  the  outage. 


Management  Actions 


Both  the  321st  Theater  Materiel  Management  Center  Rear  and  the  377th  Theater 
Support  Command  issued  after-action  reports  following  Hurricane  Katrina  that 
documented  the  communications  outages,  the  actions  taken  to  restore 
communications,  lessons  learned  from  the  event,  and  recommendations  to  prepare 
for  future  disasters.  The  32 1st  Theater  Materiel  Management  Center  Rear  took 
the  following  steps  to  correct  the  identified  weaknesses  in  disaster  preparedness: 

•  developed  a  formal,  approved  disaster  recovery  plan  that  meets  the 
requirements  established  in  DoD  Directive  3020.26  and  DoD 
Instruction  8500.2; 
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•  tested  the  disaster  recovery  plan  as  required  by  DoD  Directive  3020.26 
and  DoD  Instruction  8500.2; 

•  added  redundant  data  communications  lines  that  were  routed  away 
from  the  Gulf  Coast  and  added  back-up  satellite  communications 
capabilities;  and 

•  implemented  training  and  certification  programs  to  prepare  users  to 
operate  on  generator  power  or  satellite  communications,  or  both. 


Mission  Transfer 


According  to  Anny  Materiel  Command  officials,  the  mission  for  the  CTASC 
system  in  Baton  Rouge  was  transferred  to  the  Army  Materiel  Command 
Logistics  Support  Activity,  Redstone  Arsenal,  Alabama,  in  June  2006.  Army 
Materiel  Command  officials  stated  that  this  move  is  the  result  of  an  Army¬ 
wide  effort,  initiated  prior  to  Hurricane  Katrina,  that  was  designed  to 
consolidate  all  nine  CTASC  systems  at  one  site  to  reduce  the  number  of 
systems,  prepare  for  implementing  future  logistics  infonnation  svstems,  and 
support  the  Army’s  transformation  to  a  modular  force  structure/ 

The  concept  plans  for  CTASC  system  consolidation,  April  25,  2006,  state  that 
a  COOP  site  will  be  located  at  the  Defense  Information  Systems  Agency 
facility  in  Oklahoma  City,  Oklahoma.  Army  Materiel  Command  officials 
stated  that  the  COOP  site  will  serve  as  a  full-time  back-up  site  to  the  CTASC 
system  operations  at  the  Logistics  Support  Activity.  The  COOP  site  was  not 
yet  operational  as  of  September  2006  because  the  Army  Materiel  Command 
and  the  Defense  Information  Systems  Agency  are  negotiating  the  fee-for- 
service  contract.  According  to  Army  Materiel  Command  officials,  only  the 
CTASC  system  located  at  the  321st  Theater  Materiel  Management  Center 
Rear  will  join  the  three  systems  already  located  at  Anny  Materiel  Command 
Logistics  Support  Activity.  The  other  six  systems  will  remain  at  their  existing 
locations  until  the  COOP  site  is  activated  because  AMC  officials  determined 
there  are  not  enough  spare  CTASC  systems  available  to  satisfy  the  program’s 
self-imposed  one-for-one  backup  rule. 

In  addition  to  the  CTASC  system’s  relocation,  officials  at  the  377th  Theater 
Support  Command  stated  during  audit  field  work  that  the  Standard  Army 
Ammunition  System-Modernized  mission  was  tentatively  scheduled  to 
transfer  to  the  377th  Theater  Support  Command  in  Kuwait  around  June  2006. 
Statements  made  by  the  Commanding  General  of  the  377th  Theater  Support 
Command  in  an  August  2006  response  to  the  draft  report  indicated  that  the 
Standard  Army  Ammunition  System-Modernized  was  transferred  to  Belle 
Chasse  in  July,  but  there  was  no  official  order  for  the  transfer:  it  was  part  of 
an  inactivation  process  and  that  the  mission  remained  with  the  377th  Theater 
Support  Command.  The  Commanding  General  stated  that  a  plan  to  transfer 


3  The  Army  is  restructuring  from  a  division-based  to  a  brigade-based  force,  which  will  result  in  Brigade  Combat 
Teams  that  are  designed  as  self-sufficient  and  standardized. 
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the  Standard  Army  Ammunition  System-Modernized  to  another  location  is 
being  analyzed.  Further,  the  Commanding  General  stated  that  although  the 
initial  plan  was  to  transfer  the  Standard  Army  Ammunition  System- 
Modernized  operations  to  Kuwait,  the  final  decision  was  to  move  it  to  Belle 
Chasse,  Louisiana.  Regardless  of  the  actual  location  of  the  primary  and 
back-up  system,  once  the  move  is  made,  the  Command  will  need  to  establish 
disaster  recovery  measures. 
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Recommendations  and  Management  Comments 


1.  We  recommend  that  the  Commander,  U.S.  Army  Materiel  Command 
execute  the  planned  disaster  recovery  measures  after  consolidating  Army 
Corps  Theater  Automatic  Data  Processing  Service  Center  systems  at  the 
Logistics  Support  Activity,  Redstone  Arsenal,  Alabama. 

Management  Comments.  The  Commander,  U.S.  Army  Materiel  Command 
concurred  with  the  recommendation,  stating  that  the  U.S.  Army  Logistics  Support 
Activity  will  develop  a  Corps  Theater  Automation  Service  Support  Center 
disaster  recovery  plan  that  complies  with  DoD  Instruction  8500.2  and  DoD 
Directive  3020.26  by  January  1,  2007.  Additionally,  the  Commander  stated  that 
current  continuity  of  operations  planning  employs  a  combination  of  CTASC 
hardware  at  the  Software  Engineering  Center,  Fort  Belvoir,  Virginia,  and  Army 
command  CTASC  in  disbursed  locations.  The  Commander  also  stated  that 
U.S.  Army  Materiel  Command  is  actively  acquiring  and  locating  100  percent 
redundant  hardware  and  software  at  the  Software  Engineering  Center,  Fort  Lee, 
Virginia,  which  will  be  activated  by  January  1,  2007. 

2.  We  recommend  that  the  Commander,  377th  Theater  Support  Command 
establish  disaster  recovery  measures  that  comply  with  Federal  and  DoD 
policy  at  the  final  locations  of  both  the  primary  and  back-up  Standard  Army 
Ammunition  System-Modernized  systems. 

Management  Comments.  The  Commanding  General  of  the  U.S.  Anny  377th 
Theater  Support  Command  concurred  with  comments  to  the  recommendation. 

The  Commanding  General  stated  that  the  Command  recognized  and  corrected  the 
deficiencies  prior  to  the  audit.  After  Hurricane  Katrina,  the  Command  developed 
and  tested  a  disaster  recovery  plan  to  ensure  continuity  of  key  operations.  They 
added  redundant  data  communications  and  trained  staff  on  this  process. 
Additionally,  the  Commanding  General  stated  the  Standard  Army  Ammunition 
System-Modernized  was  transferred  to  Belle  Chasse  on  July  12,  2006,  and  the 
Command  was  in  the  process  of  modifying  the  disaster  recovery  plan  to  include 
the  system  at  that  location.  The  plan  will  include  disaster  recovery  measures  that 
comply  with  Federal  and  DoD  policy  at  the  final  locations  of  both  the  primary 
and  back-up  Standard  Anny  Ammunition  System-Modernized  systems.  Finally, 
the  Commanding  General  stated  that  the  new  plan  will  be  completed  on  or  before 
November  30,  2006. 
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Appendix  A.  Scope  and  Methodology 


Scope  and  Methodology.  We  conducted  this  audit  from  February  through  July 
2006  in  accordance  with  generally  accepted  government  auditing  standards.  DoD 
facilities  in  Louisiana,  Mississippi,  Alabama,  and  Florida  were  damaged  by 
Hurricane  Katrina.  We  determined  that  Army  logistics  information  technology 
resources  were  affected  in  Baton  Rouge,  Louisiana.  We  reviewed  the  information 
technology  resources  at  the  321st  Theater  Materiel  Management  Center  Rear, 
located  at  the  Leonard  C.  Saurage  U.S.  Army  Reserve  Center.  The  321st  Theater 
Materiel  Management  Center  Rear  uses  the  CTASC  system  and  the  Standard 
Anny  Ammunition  System-Modernized  to  execute  its  mission,  and  both  systems 
suffered  communications  disruptions.  We  conducted  field  work  and  interviewed 
personnel  at  the  Department  of  the  Anny,  Deputy  Chief  of  Staff,  G-4  (Logistics); 
the  Office  of  the  Army  Chief  Information  Officer  (CIO/G-6);  the  U.S.  Army 
Materiel  Command;  377th  Theater  Support  Command,  located  at  the  Naval  Air 
Station,  Joint  Reserve  Base,  New  Orleans,  Louisiana;  and  the  321st  Theater 
Materiel  Management  Center  Rear,  Baton  Rouge,  Louisiana.  Additionally,  we 
reviewed  and  analyzed  disaster  recovery  plans  and  procedures  in  place  before  and 
after  Hurricane  Katrina  to  determine  whether  they  complied  with  Federal  and 
DoD  disaster  recovery  planning  regulations  and  instructions. 

Use  of  Computer-Processed  Data.  We  did  not  use  computer-processed  data  to 
perform  this  audit. 

Government  Accountability  Office  High-Risk  Area.  The  Government 
Accountability  Office  (GAO)  has  identified  several  high-risk  areas  in  DoD.  This 
report  provides  coverage  of  the  Protecting  the  Federal  Government’s  Information 
Systems  and  the  Nation’s  Critical  Infrastructures  high-risk  areas. 


Prior  Coverage 


During  the  last  5  years,  GAO  has  issued  one  report  on  continuity  planning  and 
emergency  recovery  efforts.  Unrestricted  GAO  reports  can  be  accessed  over  the 
Internet  at  http  ://ww w.  gap .  gov. 

GAO  Report  GAO-04- 160,  “Continuity  of  Operations:  Improved  Planning 
Needed  to  Ensure  Delivery  of  Essential  Government  Services,”  February 
27, 2004 
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Appendix  B.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Director,  Acquisition  Resources  and  Analysis 
Under  Secretary  of  Defense  (Comptroller)/Chief  Financial  Officer 
Under  Secretary  of  Defense  for  Personnel  and  Readiness 
Assistant  Secretary  of  Defense  for  Networks  and  Infonnation  Integration/  Chief 
Infonnation  Officer 

Director,  Program  Analysis  and  Evaluation 


Joint  Staff 

Director,  Joint  Staff 


Department  of  the  Army 

Commander,  U.S.  Anny  Materiel  Command 
Deputy  Chief  of  Staff,  G-4  (Logistics) 

Anny  Chief  Information  Officer  (CIO/G-6) 

Commander,  U.S.  Army  Reserve  Command 
Commander,  377th  Theater  Support  Command 
Commander,  321st  Theater  Materiel  Management  Center 
Auditor  General,  Department  of  the  Anny 


Combatant  Command 

Commander,  U.S.  Central  Command 


Other  Defense  Organizations 

Director,  Defense  Information  Systems  Agency 
Director,  Defense  Logistics  Agency 
Director,  National  Guard  Bureau 


Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget 

Office  of  Inspector  General,  Department  of  Homeland  Security 
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Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 

Senate  Committee  on  Homeland  Security  and  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Anned  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Management,  Finance,  and  Accountability 
Committee  on  Government  Refonn 

House  Subcommittee  on  National  Security,  Emerging  Threats,  and  International 
Relations,  Committee  on  Government  Reform 
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U.S.  Army  Materiel  Command  Comments 


DEPARTMENT  OF  THE  ARMY 

HEADQUARTERS,  U.S.  ARMY  MATERIEL  COMMAND 
9301  CHAPEK  ROAD 
FORT  BELVOIR,  VA  22060-5627 


AMCIR  I  August  2006 


MEMORANDUM  FOR  Mr.  David  Lawson,  Chief,  The  Army  Audit  Liaison  Office,  U.S.  Army 
Audit  Agency,  3101  Park  Center  Drive,  Alexandria,  VA  22302-1506 

SUBJECT:  DODJG  Draft  Report,  Hurricane  Katrina  Disaster  Recovery  Efforts  Related  to  Army 
Information  Technology  Resources,  D2Q06-DO00AS-0135.000,  AMC  D0617 


1.  Headquarters,  U.S.  Army  Materiel  Command  (HQ  AMC)  has  reviewed  the  subject  draft 
report  and  agree  with  the  reported  facts,  conclusions  and  recommendation.  Specific  comments 
are  enclosed. 

2.  HQ  AMC  point  of  contact  is  Mr.  Tilden  Jio,  703-806-9021,  email:  tilden.jio@us.army.mil. 
FOR  THE  COMMANDER: 


Enel  SUSAN  C.  McCOY 

Director,  Internal  Review  and 
Audit  Compliance  Office 

CF  (w/encl): 

IG,  DOD,  400  Army  Navy  Drive,  Arlington,  VA  22202-4704 
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Command  Comments 

DODIG  Draft  Report,  Project  Number  D2O06-DOOOAS-O135.OOO 
Hurricane  Katrina  Disaster  recovery  Efforts  Related  to  Army  Information  Technology  Resources 

Objective:  The  overall  objective  was  to  determine  whether  adequate  internal  controls  and  plans 
for  disaster  recovery  were  in  place  to  safeguard  Army  information  technology  resources. 

Conclusion:  The  321s'  Theater  Materiel  Management  Center  in  Baton  Rouge  experiences 
communication  outages  as  a  result  of  Hurricane  Katrina  that  adversely  affected  the  operations  of 
the  CTASC  and  Standard  Army  Ammunition  System-Modernized  Systems.  The  321"  Theater 
Materiel  management  Center  did  not  adequately  plan  for  disaster  recovery  prior  to  Hurricane 
Katrina  and  was  forced  to  develop  solutions  to  the  communication  outages  as  they  occurred. 
Following  Hurricane  Kartina,  the  321”  Theater  Materiel  Management  Center  developed  and 
implemented  plans,  procedures,  and  equipment  upgrades  to  prepare  for  future  drsasters.  During 
the  audit  field  work.  Army  Materiel  Command  officials  provided  documents  lliat  stated  the 
mission  for  the  CTASC  system  at  the  321“  Theater  Materiel  Management  Center  was  transferred 
to  the  Army  Materiel  Command  in  June  2006.  Additionally,  the  377th  Theater  Support 
Command  changed  the  location  where  the  Standard  Army  Ammunition  System-Modernized 
would  he  located,  but  could  not  provide  official  orders  directing  the  move. 

Recommendation  1:  We  recommend  that  the  Commander,  U.S.  Army  Materiel  Command 
execute  the  planned  disaster  recovery  measures  after  consolidating  Army  Corps  Theater 
Automatic  Data  Processing  Sendee  Center  systems  at  the  Logistics  Support  Activity,  Redstone 
Arsenal,  Alabama. 

Command  Comments:  Concur.  US  Army  Logistics  Support  Activity  (LOGSA)  will  develop  a 
Corps  Theater  Automation  Service  Support  Center  (CTASC)  disaster  recovery  plan  in 
accordance  with  DoD  Instruction  8500.2  and  DoD  Directive  3020.26  not  later  than  1  Jan  07:  the 
plan  will  be  exercised  at  least  annually  and  updated  semiannually.  Disaster  recovery  plans  are 
phased  as  follows.  Current  Continuity  of  Operations  planning  employs  a  combination  of 
CTASC  hardware  at  Software  Engineering  Center  Fort  Belvoir.  VA  (SEC  Belvoir)  and  Army 
Command  CTASC  in  disbursed  locations.  Not  later  than  1  Jan  07,  we  will  have  an  operational 
disaster  recover  site.  We  arc  actively  acquiring  and  locating  one  hundred  percent  redundant 
hardware  and  software  at  Software  Engineering  Center  Fort  Lee,  VA  (SEC  Lee). 


Enclosure 
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U.  S.  Army  377th  Theater  Support  Command 
Comments 


REPLY  TO 
ATTENTION  OF: 


DEPARTMENT  OF  THE  ARMY 

HEADQUARTERS,  377™  THEATER  SUSTAINMENT  COMMAND 
400  RUSSELL  AVENUE,  BLDG  492 
BELLE  CHASSE  NAS,  LOUISIANA  70143-5077 


AFRC-SBLA-1R 


29  August  2006 


MEMORANDUM  FOR  Department  of  Defense,  Office  of  the  Inspector  General, 
Acquisition  and  Contract  Management  Directorate 
400  Army  Navy  Drive 
Arlington,  Virginia  22202-4704 

SUBJECT:  Management  Comments  to  DoDIG  Draft  Audit  Report  on  Hurricane  Katrina 
Disaster  Recovery  Efforts  Related  to  Army  Information  Technology  Resources 


1 .  Per  discussions  with  the  Audit  Team,  our  management  comments  to  the  Draft  Audit  Report 
on  Hurricane  Katrina  Disaster  Recovery  Efforts  Related  to  Army  Information  Technology 
Resources,  Project  No.  D2006-D000AS-01 35.000,  have  been  revised  to  include  an  additional 
statement  in  the  Management  Comments  to  Recommendation  2  and  delete  paragraph  I  in  the 
Corrected  Comments  to  the  Draft  Audit  Report  section.  The  revised  comments  are  provided  at 
the  Enclosure  1  and  the  original  comments  are  at  Enclosure  2  for  your  convenience. 

2.  The  point  of  contact  for  this  action  is  Ms.  Quynh  Dang,  (504)  552-5869,  e-mail: 
quynh.dang@usar.army.mil. 


2  Ends 


9, 


^C^AHARRY  J  PHILIPS,  JR 
(J  Major  General,  USAR 
Commanding  General 


Cc)C 


*  Omitted  Enclosure  2  because  of  length. 


Final  Report 
Reference 


* 
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Hurricane  Katrina  Dijgfjer  Recovery  Efforts  Related  to  Army  Information 

Technology  Resource!! 

Project  No.  D20O6-DOOOAS-Q 1 35.000 


RFCOMMFNDATION/MANAOFMFNT  COMMENTS 

Recommendation  2.  The  Commander,  377lh  Theater  Support  Command,  establish  disaster 
recovery  measures  that  comply  with  Federal  and  Department  of  Defense  policy  at  the  final 
locations  of  both  the  primary  and  backup  Standard  Army  Ammunition  System-Modernized 
systems. 

Management  Comments.  Concur  with  comments.  Deficiencies  stated  in  the  report  were 
recognized  and  corrected  by  our  management  prior  to  the  audit.  After  hurricane  Katrina,  we 
identified  the  need  of  having  a  disaster  recovery  plan  in  place  to  ensure  continuity  of  our  key 
operations.  A  disaster  recovery  plan  was  developed  and  tested,  redundant  data  communications 
were  added,  and  necessary  training  was  provided  to  staff.  The  Standard  Army  Ammunition 
System-Modernized  (SAAS-MOD)  was  transferred  to  Belle  Chasse  on  July  12  and  we  are  in 
the  process  of  modifying  the  disaster  recovery  plan  to  include  SAAS-MOD  at  the  new  location. 
The  new  plan  will  be  completed  on  or  before  30  November  2006  and  will  include  disaster 
recovery  measures  that  comply  with  Federal  and  Department  of  Defense  policy  for  both 
primary  and  backup  SAAS-MOD  systems. 


I 

Enclosure  1 
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Final  Report 
Reference 


Hurricane  Katrina  Disaster  Recovery  Efforts  Related  lo  Army  Information 

Technology  Resources 
I'roiect  No.  D2OO6-DOOOAS-O135.0OQ 


CORRI  (III)  COMMENTS  TO  THE  AUDI  I  DRAM  RLI’OR  I 


Incorrect  Data.  Paragraph  “Mission  Transfer"  on  page  6  of  the  report  states,  . .  Statements 
made  by  a  377lh  Theater  Support  Command  internal  review  evaluator  in  a  July  2006  response 
to  a  discussion  draft  of  this  report  indicated  that  the  primary  Standard  Army  Ammunition 
System-Modernized  operations  in  Southwest  Asia  will  be  conducted  at  the  377lh  Theater 
Support  Command  Headquarters  in  Belle  Chasse,  Louisiana,  instead  of  Kuwait.  Further,  the 
internal  evaluator  stated  that  a  backup  system  will  be  located  at  the  321“  Theater  Materiel 
Management  Center  in  Baton  Rouge;  however,  the  internal  evaluator  could  not  provide  any 
official  orders  directing  this  move." 

Corrected  Data.  There  was  no  official  order  for  the  transfer  of  SAAS-MOD.  The  transfer 
was  part  of  an  inactivation  process.  The  USAR  issued  Permanent  Orders  directing  the 
inactivation  of  321"  TMMC  with  effective  date  1 5  September  2006.  321s'  TMMC  is 
subordinate  to  377,h  TSC.  Although  the  unit  was  being  inactivated,  the  mission  remained  with 
377lh  TSC.  Therefore,  a  plan  to  transfer  SAAS-MOD  to  another  location  was  being  analyzed. 
The  initial  plan  was  to  transfer  SAAS-MOD  operations  to  Kuwait.  However,  final  decision 
was  made  lo  move  SAAS-MOD  to  Belle  Chasse,  Louisiana. 


2 


Enclosure  1 


Revised 
Page  5 
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